Skip to content

fix(taskblock): reject invalid thresholds and recover abandoned intervals (PROF-14931) - #915

Open
jbachorik wants to merge 38 commits into
paul.fournillon/taskblock-corefrom
taskblock-review-fixes
Open

jbachorik wants to merge 38 commits into
paul.fournillon/taskblock-corefrom
taskblock-review-fixes

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Hardens the paired TaskBlock API added in #664:

  • Fails closed when the 1 ms threshold derived from the TSC frequency is zero. These intervals count in task_block_threshold_invalid, not in task_block_skipped_too_short.
  • beginTaskBlock() recovers an interval whose endTaskBlock() never ran. Recovery runs only after the begin admission checks, and it closes the tracker run only when the token matches the current slot and generation. It counts in task_block_recovered_abandoned.
  • JVMSupport::isPlatformThread() returns false on non-HotSpot VMs that report JNI 19 or later, without reading the virtual-thread slot of the JNI function table.
  • Counts a blockEnter skipped by an active trace context in block_enter_skipped_trace_context.

Motivation:

An uncalibrated clock (TSC frequency below 1 kHz) makes the threshold zero. A zero threshold would admit every positive interval and hide the clock problem as normal events.

A missed endTaskBlock() must not leave the thread's paired token in place forever, because that also suppresses its later wall signals. Native code cannot tell an abandoned begin from a nested one, so the replacement rule must be explicit.

IsVirtualThread is read through a hard-coded JNI table offset (slot 234). This offset is verified only on HotSpot. Reading it on another VM could call the wrong function.

Additional Notes:

  • Stack: builds on feat(taskblock): record completed intervals with DEBUG diagnostic receipts (PROF-14931) #664. The clock change validates the derived threshold. It does not change TSC calibration.
  • Reviewer acknowledgment, paired API semantics: a begin that passes admission discards any pending interval without an event, nested or abandoned. A later endTaskBlock() with the discarded token returns false. Begin keeps the pending interval when it returns 0 for these early reasons: recording inactive, caller not verified, not a platform thread, or active span. A later failure to arm the replacement can also return 0 after the discard, so 0 alone does not mean the old interval was kept. Callers must still call endTaskBlock() in finally; the Javadoc says so.
  • Under an active span, recovery waits: a nested live interval is indistinguishable from an abandoned one, and keeping the live one has priority.
  • Reviewer acknowledgment, VM support: on non-HotSpot VMs with JNI 19 or later, beginTaskBlock() always returns 0 and endTaskBlock() returns false. This restricts the paired API only, not other profiling. feat(taskblock): add JVM producers and retain failed recording output (PROF-14931) #665 extends the classifier contract to the JVM producers.
  • No new JFR field, automatic producer, backend change or rollout.
  • Contracts: paired intervals, platform-thread classification.
  • Human review: mandatory (label human_review:mandatory). Reason: changes the public beginTaskBlock() contract (silent discard of a pending interval) and JVM-specific classification.

How to test the change?:

Added coverage:

  • gtests: TaskBlockRecorderTest.UncalibratedTscFailsClosed and UninitializedThresholdIsRecomputedFromTscFrequency; AbandonedTaskBlockTest (abandoned run closed, nothing pending, non-decodable token, newer run and foreign slot not closed); TaskBlockRealRecorderTest (stripe release and single count on stack failure, contention, inactive recorder, thread-table overflow); PairedTaskBlockRestartTest; JvmSupportNonHotspotTest (JNI 19 and 21 rejected without reading the table); ProfilerLockIndexTest; WallClockInitialEpochTest; WallTailRecordingTest.TailCarriesWeightAccumulatedWhileBlocked.
  • Java: JavaProfilerTaskBlockApiTest (abandonedIntervalDoesNotDisableTaskBlockForThread, traceContextRejectionKeepsOpenInterval), JavaProfilerTaskBlockNonHotspotTest.platformThreadCannotArmTaskBlock (fail-closed classification, no event), UnfilteredWallPrecheckTest.tracedBlockEnterIsSkippedAndCounted, JavaProfilerTaskBlockLightweightTest.

Run:

  • ./.claude/commands/build-and-summarize :ddprof-lib:gtestDebug :ddprof-lib:gtestRelease
  • ./.claude/commands/build-and-summarize :ddprof-test:testDebug -Ptests='JavaProfilerTaskBlock*'
  • JavaProfilerTaskBlockNonHotspotTest needs a non-HotSpot JDK 19+ as JAVA_TEST_HOME; it runs only where platform-thread classification is not supported.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review (run the dd:platform-security-review skill, or file a request via the PSEC review form). bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-14931

Unsure? Have a question? Request a review!

A begin whose endTaskBlock was never reached left the TLS token set,
so every later begin on the thread failed, and its TASK_BLOCK run kept
suppressing wall-clock signals. A new begin now supersedes the pending
interval: it clears the token and closes the run with the
generation-checked exitBlockedRun(), counting
TASK_BLOCK_RECOVERED_ABANDONED. Document the finally requirement and
the supersede policy on beginTaskBlock.
isPlatformThread() returned true on non-HotSpot JVMs reporting JNI 19+
without probing, so virtual threads could enter task-block producer
state. The hardcoded IsVirtualThread table offset is only trusted on
HotSpot, so other VMs now report every thread as non-platform, which
keeps task-block hooks disabled there. Skip the arming TaskBlock tests
on those JVMs and add a test asserting begin returns 0 there.
computeMinTaskBlockTicks() yields 0 when TSC::frequency() is below
1 kHz, and a zero threshold let every block pass the 1 ms minimum.
exceedsMinTaskBlockDuration() now rejects in that case, which the
caller counts as TASK_BLOCK_SKIPPED_TOO_SHORT. Add test-only hooks to
reset the cached threshold and replace the frequency source, and
cover the lazy recompute path and the fail-closed path.
Add TaskBlockAssertions.assertContainsDuration() and bound the encoded
duration in the paired-API and lightweight TaskBlock tests.
Add a test-only observer for the event recordUnownedBlockedTail()
writes and a gtest that accumulates suppressed weight before the tail
flush and checks the emitted weight matches it.
blockEnter0 returned 0 for traced threads without any counter, unlike
beginTaskBlock0. Increment BLOCK_ENTER_SKIPPED_TRACE_CONTEXT on that
path and test it.
@jbachorik jbachorik added the AI label Oct 27, 2026
@jbachorik
jbachorik marked this pull request as ready for review October 27, 2026 18:03
@jbachorik
jbachorik requested a review from a team as a code owner October 27, 2026 18:03
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T18:06:58.404891Z b626adf Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Abandoned TaskBlock recovery retains generation checks, and invalid duration thresholds fail closed. The completed static review reported no actionable correctness regression; runtime behavior was not exercised.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit b626adf

@dd-octo-sts

dd-octo-sts Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #44815006780 | Commit: 48eb4b2 | Duration: 0s (longest job)

✅ All 0 test jobs passed

No CI outcome reports were found for this run.
Summary: Total: 0 | Passed: 0 | Failed: 0


Updated: 2026-11-20 20:54:03 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 3af96ec2

@datadog-datadog-prod-us1

This comment has been minimized.

@kaahos
kaahos added this pull request to stack #988 November 12, 2026 23:32
Emit the getLockIndex definition for cross-translation-unit callers and
cover optimized linking with a recording-stripe lookup regression.

Wait for the wall timer's initial sampling epoch in the debug-only
synchronous dump driver. Bound the normal-thread wait and test ready,
delayed-publication, and timeout behavior without changing zero-epoch
production fallback or weakening the JFR assertions.
@dd-octo-sts

dd-octo-sts Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Reliability & Chaos Results

✅ All reliability & chaos checks passed Pipeline: https://gitlab.ddbuild.io/DataDog/java-profiler/-/pipelines/153120055

@kaahos
kaahos removed this pull request from stack #988 November 14, 2026 10:46
Do not schedule musl/OpenJ9 integration jobs: upstream publishes no musl runtime. Report these combinations as N/A while retaining failure and completeness checks for applicable configurations.

Applies to #663.
@kaahos kaahos changed the title fix(taskblock): reject invalid thresholds and recover abandoned intervals (PROF-16669) fix(taskblock): reject invalid thresholds and recover abandoned intervals (PROF-14931) Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

The PR hardens TaskBlock recovery, preserves intervals after early rejection, and fails closed on unsupported classification and invalid thresholds. One issue remains: the new trace-context preservation test can fail intermittently when recorder contention rejects completion; it needs the bounded-retry helper already used by neighboring tests.

Open Bits AI session

🤖 Bits Code Review · Commit 0260ef4

Comment on lines +206 to +223
AtomicLong nested = new AtomicLong(-1L);
AtomicBoolean recorded = new AtomicBoolean();
runWorker(
() -> {
long token = profiler.beginTaskBlock();
assertTrue(token != 0);
profiler.setTraceContext(0x5200L, 0x5201L, 0L, 0x5201L, -1, null, -1, null);
try {
nested.set(profiler.beginTaskBlock());
} finally {
profiler.clearTraceContext();
}
Thread.sleep(200L);
recorded.set(profiler.endTaskBlock(token, BLOCKER, UNBLOCKING_SPAN_ID));
});
assertEquals(0L, nested.get());
assertTrue(
recorded.get(), "a begin rejected for an active trace context must keep the open interval");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Retry recorder refusals in the trace-context preservation test

The wall timer can close an epoch concurrently with this completion, holding the recorder lock exclusively. endTaskBlock can then legitimately return false because recording is best-effort, making this assertion fail even though the interval was correctly preserved. Wrap the scenario in recordWithinAttempts so legitimate recorder refusals are retried and checked against task_block_record_failed.

Suggested change
AtomicLong nested = new AtomicLong(-1L);
AtomicBoolean recorded = new AtomicBoolean();
runWorker(
() -> {
long token = profiler.beginTaskBlock();
assertTrue(token != 0);
profiler.setTraceContext(0x5200L, 0x5201L, 0L, 0x5201L, -1, null, -1, null);
try {
nested.set(profiler.beginTaskBlock());
} finally {
profiler.clearTraceContext();
}
Thread.sleep(200L);
recorded.set(profiler.endTaskBlock(token, BLOCKER, UNBLOCKING_SPAN_ID));
});
assertEquals(0L, nested.get());
assertTrue(
recorded.get(), "a begin rejected for an active trace context must keep the open interval");
recordWithinAttempts(
() -> {
AtomicLong nested = new AtomicLong(-1L);
AtomicBoolean recorded = new AtomicBoolean();
runWorker(
() -> {
long token = profiler.beginTaskBlock();
assertTrue(token != 0);
profiler.setTraceContext(0x5200L, 0x5201L, 0L, 0x5201L, -1, null, -1, null);
try {
nested.set(profiler.beginTaskBlock());
} finally {
profiler.clearTraceContext();
}
Thread.sleep(200L);
recorded.set(profiler.endTaskBlock(token, BLOCKER, UNBLOCKING_SPAN_ID));
});
assertEquals(0L, nested.get());
return recorded.get();
});

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

kaahos added a commit that referenced this pull request Nov 20, 2026
@kaahos kaahos added the human_review:mandatory Human review required — see PR description label Nov 20, 2026

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The PR hardens TaskBlock recovery and admission for abandoned intervals, active trace contexts, unsupported classification, and invalid thresholds. The earlier flaky assertion is fixed by bounded retries that verify recorder-failure accounting while still detecting lost intervals; no findings remain open.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit dad1920

kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026
kaahos added a commit that referenced this pull request Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI human_review:mandatory Human review required — see PR description

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants